Fix getdeps fallback mirror downloads#14763
Conversation
|
| Check | Count |
|---|---|
concurrency-mt-unsafe |
9 |
cppcoreguidelines-avoid-non-const-global-variables |
5 |
cppcoreguidelines-pro-type-member-init |
1 |
| Total | 15 |
Details
db/db_wal_test.cc (1 warning(s))
db/db_wal_test.cc:117:5: warning: uninitialized record type: 'fs_stat' [cppcoreguidelines-pro-type-member-init]
db_stress_tool/db_stress_common.cc (4 warning(s))
db_stress_tool/db_stress_common.cc:22:25: warning: variable 'raw_env' is non-const and globally accessible, consider making it const [cppcoreguidelines-avoid-non-const-global-variables]
db_stress_tool/db_stress_common.cc:22:25: warning: variable 'raw_env' provides global access to a non-const object; consider making the pointed-to data 'const' [cppcoreguidelines-avoid-non-const-global-variables]
db_stress_tool/db_stress_common.cc:25:56: warning: variable 'wbm' is non-const and globally accessible, consider making it const [cppcoreguidelines-avoid-non-const-global-variables]
db_stress_tool/db_stress_common.cc:26:49: warning: variable 'rate_limiter' is non-const and globally accessible, consider making it const [cppcoreguidelines-avoid-non-const-global-variables]
db_stress_tool/db_stress_test_base.cc (6 warning(s))
db_stress_tool/db_stress_test_base.cc:261:7: warning: function is not thread safe [concurrency-mt-unsafe]
db_stress_tool/db_stress_test_base.cc:4025:7: warning: function is not thread safe [concurrency-mt-unsafe]
db_stress_tool/db_stress_test_base.cc:4566:7: warning: function is not thread safe [concurrency-mt-unsafe]
db_stress_tool/db_stress_test_base.cc:4600:11: warning: function is not thread safe [concurrency-mt-unsafe]
db_stress_tool/db_stress_test_base.cc:4639:11: warning: function is not thread safe [concurrency-mt-unsafe]
db_stress_tool/db_stress_test_base.cc:5373:7: warning: function is not thread safe [concurrency-mt-unsafe]
db_stress_tool/db_stress_tool.cc (4 warning(s))
db_stress_tool/db_stress_tool.cc:41:5: warning: variable 'fault_fs_for_crash_report' is non-const and globally accessible, consider making it const [cppcoreguidelines-avoid-non-const-global-variables]
db_stress_tool/db_stress_tool.cc:402:7: warning: function is not thread safe [concurrency-mt-unsafe]
db_stress_tool/db_stress_tool.cc:416:9: warning: function is not thread safe [concurrency-mt-unsafe]
db_stress_tool/db_stress_tool.cc:448:9: warning: function is not thread safe [concurrency-mt-unsafe]
|
@xingbowang has imported this pull request. If you are a Meta employee, you can view this in D105859558. |
Codex Code Review - OBSOLETESuperseded by a newer AI review. Expand to see the original review.🟡 Codex Code ReviewAuto-triggered after CI passed — reviewing commit 556211b ❌ Codex review failed before producing findings. ℹ️ About this responseGenerated by Codex CLI. Limitations:
Commands:
|
Claude Code Review - OBSOLETESuperseded by a newer AI review. Expand to see the original review.✅ Claude Code ReviewAuto-triggered after CI passed — reviewing commit 556211b SummaryThis PR fixes real CI failures in the folly getdeps fallback mirror script by addressing manifest parsing, atomic downloads, cache validation, and adding libiberty. The changes are well-targeted and improve reliability and security. Most agent-flagged issues were resolved as false positives or theoretical concerns during debate. High-severity findings (0): No high-severity findings. Full review (click to expand)Findings🔴 HIGHNone after debate resolution. Three initially-flagged HIGH findings were withdrawn:
🟡 MEDIUMM1. Verify manifest files only contain GNU package URLs --
|
| Context | Applicable? | Assessment |
|---|---|---|
| folly.mk integration | Yes | Compatible. Script output/exit code behavior preserved. |
| getdeps.py interaction | Yes | File naming {package}-{filename} matches existing convention. |
| wget removal | Yes | Net improvement. urllib.request has better defaults for SSL, redirects, proxies. |
| Concurrent execution | No | Script called from single make target; never runs concurrently. |
| Windows/macOS | No | folly build in this Makefile is Linux-only. |
Assumption stress-test results:
- "Atomic downloads" claim: Holds for single-process POSIX usage. SIGKILL can leave .tmp files, but these are cleaned up on next run.
- "Bad files are validated and removed" claim: Holds.
sha256_file()returningNoneon I/O error means the file is treated as non-existent (not validated), which is safe -- the download path will be attempted. - "urllib is suitable wget replacement" claim: Holds. urllib handles redirects (important for ftpmirror.gnu.org), validates SSL, and respects proxy env vars.
Positive Observations
- Fixes the actual root cause of CI failures (empty files from wget + ConfigParser parsing).
- Atomic temp-file download pattern prevents partial file persistence.
- Cache validation removes bad cached files that caused repeated failures.
- Removes external
wgetdependency in favor of Python stdlib. - Fixes ConfigParser interpolation vulnerability (
interpolation=None). - Better error messages with SHA256 values and file sizes for debugging.
- Adding
libibertyto the package list addresses a known missing dependency.
ℹ️ About this response
Generated by Claude Code.
Review methodology: claude_md/code_review.md
Limitations:
- Claude may miss context from files not in the diff
- Large PRs may be truncated
- Always apply human judgment to AI suggestions
Commands:
/claude-review [context]— Request a code review/claude-query <question>— Ask about the PR or codebase
|
@xingbowang has imported this pull request. If you are a Meta employee, you can view this in D105859558. |
Codex Code Review - OBSOLETESuperseded by a newer AI review. Expand to see the original review.🟡 Codex Code ReviewAuto-triggered after CI passed — reviewing commit 075b120 ❌ Codex review failed before producing findings. ℹ️ About this responseGenerated by Codex CLI. Limitations:
Commands:
|
Claude Code Review - OBSOLETESuperseded by a newer AI review. Expand to see the original review.✅ Claude Code ReviewAuto-triggered after CI passed — reviewing commit 075b120 SummarySolid bug fix that correctly addresses the CI failure (empty file accepted due to manifest parse failure + no invalid file cleanup). The core changes -- ConfigParser fix, atomic temp-file downloads, SHA256 validation with cleanup -- are well-engineered. A few medium-severity issues around edge cases and one behavioral subtlety worth noting. High-severity findings (0): No high-severity findings. Full review (click to expand)Findings🟡 MEDIUMM1. No download size limit in
|
| Context | Affected? | Assessment |
|---|---|---|
| folly.mk cache restore (lines 126-128) | Yes | cp -n (no-clobber) won't overwrite files the script downloaded. Compatible. |
| folly.mk cache update (lines 134-139) | Yes | cp -n won't overwrite cache files the script wrote. Compatible. |
| getdeps.py fetch (line 132) | Indirect | Script pre-populates download dir; getdeps finds files by name. Filename format {package}-{filename} preserved. Compatible. |
| GitHub Actions CI | Primary | Isolated VMs, no concurrent access. All issues are mitigated. |
| Self-hosted runners | Possible | Shared /tmp could cause cache races (M4). Low practical risk. |
Positive Observations
- Atomic downloads via temp files (
download_url()) is a significant reliability improvement -- partial downloads can no longer corrupt the download directory. - Invalid file cleanup directly fixes the root cause of the CI failure (stale empty file persisted across retries).
shutil.copy2instead ofsubprocess.run(['cp', ...])removes thecpbinary dependency, improving portability.urllib.requestinstead ofwgetremoves an external tool dependency, making the script self-contained Python.- Good error messages with SHA256 values and file sizes aid debugging.
- Graceful degradation -- individual package failures are warnings, not fatal errors.
ℹ️ About this response
Generated by Claude Code.
Review methodology: claude_md/code_review.md
Limitations:
- Claude may miss context from files not in the diff
- Large PRs may be truncated
- Always apply human judgment to AI suggestions
Commands:
/claude-review [context]— Request a code review/claude-query <question>— Ask about the PR or codebase
|
@xingbowang has imported this pull request. If you are a Meta employee, you can view this in D105859558. |
🟡 Codex Code ReviewAuto-triggered after CI passed — reviewing commit 012954a ❌ Codex review failed before producing findings. ℹ️ About this responseGenerated by Codex CLI. Limitations:
Commands:
|
✅ Claude Code ReviewAuto-triggered after CI passed — reviewing commit 012954a SummaryWell-structured fix for CI download failures caused by unreliable GNU mirrors. The rewrite replaces High-severity findings (0): No high-severity findings. Full review (click to expand)Findings🟡 MEDIUMM1. No automated tests —
|
| Interaction | Status | Notes |
|---|---|---|
| getdeps.py compatibility | OK | Filename format {package}-{basename} is unchanged from working old code |
| GHA cache interaction | OK | New code validates cache entries via SHA256 and removes invalid ones in prepare_download() |
| build_folly cleanup | OK | $(restore_folly_getdeps_downloads) correctly placed after rm -rf cleanup, before getdeps.py build |
| Concurrent builds | Acknowledged | Code comment documents cache is not safe for concurrent access; acceptable for CI |
| Makefile macros | OK | define/endef blocks correctly replicate the inline shell logic |
Positive Observations
- Root cause fix: Atomic downloads via temp files directly address the original SHA256 mismatch from partial/empty downloads.
- Self-healing cache: Validates and removes corrupt cached archives, preventing persistent CI failures from bad cached data.
- Manifest parsing fix:
allow_no_value=Trueandinterpolation=Nonecorrectly handle folly manifests with bare keys that caused silent parse failures. - Improved observability: Download sizes, SHA256 values, and per-file status are logged, making CI failures much easier to debug.
- Elimination of external dependency: Replacing
wget/cpsubprocesses with Python stdlib removes a runtime dependency. - Clean Makefile refactoring: Extracting macros reduces duplication and centralizes the cache directory path.
- Correct
build_follyfix: Adding$(restore_folly_getdeps_downloads)after the cleanup step ensures archives are available whengetdeps.py buildruns.
ℹ️ About this response
Generated by Claude Code.
Review methodology: claude_md/code_review.md
Limitations:
- Claude may miss context from files not in the diff
- Large PRs may be truncated
- Always apply human judgment to AI suggestions
Commands:
/claude-review [context]— Request a code review/claude-query <question>— Ask about the PR or codebase
|
@xingbowang merged this pull request in 638354e. |
Summary:
Context:
Nightly test failed with dependency download failure in folly.
Test Plan: